IMAGE-TAKING APPARATUS AND METHOD FOR ADDING 
ANNOTATION INFORMATION TO A CAPTURED IMAGE 

[0001] This application is based on Japanese Patent Application No. 2003-186119 filed 
on June 30, 2003, the contents of which are hereby incorporated by reference. 

BACKGROUND OF THE INVENTION 

Field of the Invention 

[0002] The present invention relates to an image-taking apparatus that permits annotation 
information to be added to a region within a captured image, and relates also to a method for 
adding annotation information to a region within a captured image. 

Description of the Prior Art 

[0003] It has been proposed to add annotation information to the whole of, or a region 
within, an image captured with a digital camera. This helps to enhance images' capability of 
conveying information and to facilitate management of images. An association between a 
given piece of annotation information and a given image, or a region within it, is created 
manually by the user. Typically, the user first selects and displays an image to which the 
user wants to add annotation information, and then set a region on the image. 

[0004] Japanese Patent Application Laid-Open No. 2002-55748 proposes a method for 
efficiently adding annotation information to a large number of images stored in an image 
database. According to this method, a plurality of icons associated with individual pieces of 
annotation information are displayed along with images, and the user is invited to drag and 
drop the icon associated with particular annotation information that the user wants to add onto 



the region on an image to which the user wants to add the particular annotation information. 
This permits an association to be created between a given image, or a region within it, and a 
given piece of annotation information. 

[0005] When annotation information is added to an image in an image-taking apparatus 
such as a digital camera, the user may want to capture an image first and then add annotation 
information to it. However, conventionally, as proposed in the patent publication mentioned 
above, annotation information can be added only to images that have already been captured 
and recorded, and no consideration is given to adding annotation information to an image that 
is going to be captured. Of course, once an image to which to add annotation information is 
captured and recorded, it is possible to add annotation information to it in the conventionally 
practiced manner. This, however, requires an extra step of selecting from among recorded 
images one to which to add annotation information, resulting in low efficiency. 

SUMMARY OF THE INVENTION 
[0006] An object of the present invention is to provide an image-taking apparatus that 
permits relevant information, such as annotation information, to be added not only to an 
image that has already been recorded but also to an image that is going to be captured, and 
thus with higher efficiency, and to provide a method for adding such information in such a 
manner. 

[0007] To achieve the above object, according to one aspect of the present invention, an 
image-taking apparatus is provided with: an image sensor that captures an image; an image 
recorder that records an image captured by the image sensor on a recording medium; a display 
that displays an image recorded on the recording medium; a first selector that selects a mode 



for adding relevant information to an image captured by the image sensor; and a second 
selector that, when the first selector selects the mode for adding relevant information, selects 
as an image to which to add relevant information either an image recorded on the recording 
medium or an image that is going to be newly captured by the image sensor. 

[0008] This image-taking apparatus permits the user to choose, according to the 
circumstances, whether to add relevant information, such as annotation information, to an 
image that has already been captured and recorded or to an image that is going to be captured. 
When the user chooses to add relevant information to an image that is going to be captured, 
the user can capture the image and then immediately add relevant information to it. This 
eliminates the need for an extra step of selecting that image from among those recorded on the 
recording medium. 

[0009] Advisably, when the second selector selects an image recorded on the recording 
medium as an image to which to add annotation information, the display displays a selection 
screen for permitting selection of an image to which to add annotation information. This 
permits the user to immediately start selecting an image, and thus helps increase efficiency. 

[0010] Advisably, also when an image selected as an image to which to add annotation 
information is deleted from the recording medium, the display displays the selection screen 
for permitting selection of an image to which to add annotation information. This prevents 
the user from performing meaningless operation by attempting to add annotation information 
to an already deleted image 

[0011] Advisably, the selection screen includes an image to which to add annotation 
information and an indication relating to the annotation information already added thereto. 



This permits the user to know whether or not annotation information has already been added 
to an image to which the user wants to add new annotation information, and permits the user 
to know the already added annotation information, if any. 

[0012] Advisably, the image-taking apparatus is further provided with: an operation 
member operated by the user; and a reproducer that, when the operation member is operated 
while an image to which to add annotation information is being displayed on the display, 
reproduces the annotation information already added to the image displayed on the display. 
This permits the user to know the already added annotation information, and thereby prevents 
the user from performing unnecessary operation by adding the same or similar annotation 
information to the same image. 

[0013] Advisably, the image-taking apparatus is further provided with: a display 
controller that, when the second selector selects an image recorded on the recording medium 
as an image to which to add annotation information, displays on the display the image 
selected last time as an image to which to add annotation information before permitting 
selection of an image to which to add annotation information. The image selected last time 
is likely to be selected again as an image to which to add annotation information, and 
therefore displaying that image as a selected image by default helps increase efficiency. 

[0014] Advisably, the image-taking apparatus is further provided with: a display 
controller that, when the second selector selects an image recorded on the recording medium 
as an image to which to add annotation information, displays on the display an image to 
which annotation information has already been added as an image to which to add annotation 
information before permitting selection of an image to which to add annotation information. 



-4- 



An image to which annotation information has already been added is likely to be selected 
again as an image to which to add new annotation information, and therefore displaying that 
image as a selected image by default helps increase efficiency. 

[0015] The annotation information may be in the form of an image that is going to be 
newly captured by the image sensor. When the user chooses to add annotation information 
to an image that is going to be captured, the user first captures that image, and then captures 
another image as annotation information. In a case where a sound recorder that records 
sounds is additionally provided, the annotation information may be in the form of a sound 
recorded by the sound recorder. 

[0016] The annotation information may be added to a region within an image displayed 
on the display. As the annotation information, different pieces of annotation information 
may by added to a plurality of regions within an image displayed on the display. 

[0017] According to another aspect of the present invention, a method for adding relevant 
information to a captured image includes the steps of: selecting a mode for adding relevant 
information to a captured image; selecting, when the mode for adding relevant information is 
selected, as an image to which to add relevant information either an image recorded on a 
recording medium or an image that is going to be newly captured; and adding relevant 
information to a selected image. 

[0018] This method permits the user to choose, according to the circumstances, whether 
to add relevant information, such as annotation information, to an image that has already been 
captured or to an image that is going to be captured. When the user chooses to add relevant 
information to an image that is going to be captured, the user can capture the image and then 
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immediately add relevant information to it. This eliminates the need for an extra step of 
selecting from among the already captured images the newest to which to add relevant 
information. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0019] This and other objects and features of the present invention will become clear from 
the following description, taken in conjunction with the preferred embodiments with reference 
to the accompanying drawings in which: 

Fig. 1 is a perspective view of a digital camera embodying the invention; 
Fig. 2 is a rear view of the digital camera; 

Fig. 3 A is a diagram showing an example of a parent image to which annotations have 
been added by the digital camera; 

Fig. 3B is a diagram showing an example of an annotation image added to the parent 
image shown in Fig. 3 A by the digital camera; 

Fig. 4 is a diagram showing the structure of file folders in the digital camera; 

Fig. 5 is a diagram schematically showing the circuit configuration of the digital 
camera; 

Fig. 6 is a flow chart showing the flow of operations performed to set the 
photographing mode when electric power starts to be supplied to the digital camera; 

Fig. 7 is a diagram showing the main menu displayed by the digital camera to permit 
selection of the photographing mode; 

Fig. 8 is a flow chart showing the flow of operations performed to change the 
photographing mode in the digital camera; 

Fig. 9 is a flow chart showing the flow of operations performed in the image 
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annotation photographing mode in the digital camera; 

Fig. 10 is a diagram showing an example of the screen displayed by the digital camera 
to permit selection of a parent image to which to add an annotation; 

Fig. 1 1 is a flow chart showing the flow of operations performed to select a parent 
image in the digital camera; 

Fig. 12 is a diagram showing an example of the warning displayed by the digital 
camera when an image unsuitable as a parent image is selected; 

Fig. 13 is a diagram showing an example of the initial screen displayed by the digital 
camera to permit setting of a region within a parent image; 

Fig. 14 is a diagram showing an example of a screen including a region having its 
position changed in the digital camera; 

Fig. 15 is a diagram showing an example of a screen including a region having its 
position and size changed in the digital camera; 

Fig. 16 is a diagram showing an example of the screen displayed by the digital camera 
to permit setting of a region within an image to which an annotation has already been added; 

Fig. 17 is a flow chart showing the flow of operations performed to set a region within 
a parent image in the digital camera; 

Fig. 18 is a flow chart showing the flow of operations performed in the sound 
annotation recording mode in the digital camera; 

Fig. 19 is a flow chart showing the flow of operations performed to end the capturing 
of an annotation image and the recording of an annotation sound in the digital camera; 

Fig. 20 is a flow chart showing the flow of operations performed when the image 
annotation photographing mode or the sound annotation recording mode is started for the first 
time after electric power starts to be supplied in the digital camera; 
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Fig. 21 is a flow chart showing the flow of operations performed when a recording 
medium is mounted while electric power is being supplied in the digital camera; 

Fig. 22 is a flow chart showing the flow of operations performed when an external 
device is disconnected while electric power is being supplied in the digital camera; 

Fig. 23 is a flow chart showing the flow of operations performed to present a quick 
view in the digital camera; 

Fig. 24 is a flow chart showing the flow of operations performed to delete a file during 
a quick view in the digital camera; 

Fig. 25 is a flow chart showing another flow of operations performed to start a quick 
view when the region setting screen is being displayed in the digital camera; 

Fig. 26 is a diagram showing another main menu displayed by the digital camera to 
permit selection of the photographing mode; 

Fig. 27 is a flow chart showing the flow of operations performed in the annotation 
photographing mode in the digital camera; 

Fig. 28 is a diagram showing the screen displayed by the digital camera to permit 
choice of whether to use as a parent image an image that is going to be captured or an image 
that has already been recorded; 

Fig. 29 is a diagram showing the screen displayed by the digital camera to permit 
choice of whether to use an image or sound as an annotation to be added to the parent image; 

Fig. 30 is a flow chart showing the flow of operations performed to set the 
photographing mode when electric power starts to be supplied to the digital camera; 

Fig. 31 is a flow chart showing the flow of operations performed to change the 
photographing mode in the digital camera; and 

Fig. 32 is a diagram showing an example of the screen displayed by the digital camera 
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to permit choice of whether to use as a parent image an image that is going to be captured or 
an image that has already been recorded and which of the already recorded images to use as a 
parent image. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0020] Hereinafter, a digital camera embodying the present invention will be described 
with reference to the drawings. The outward appearance of the digital camera 100 
embodying the invention is shown in a perspective view in Fig. 1 and in a rear view in Fig. 2. 
The digital camera 100 has, on the top face thereof, a power button 1 and a shutter button 2, 
has, on the front face, a viewfinder front window 3, a microphone 4, and a taking lens 5, has, 
on a side face, a slot 7 for mounting a recording medium 6 and an external device connection 
terminal 8, and has, on the rear face, a viewfinder rear window 1 1, a mode selection dial 12, 
four direction keys 13, a liquid crystal display 14, five operation buttons 15, 16, 17, 18, and 
20, and a loudspeaker 19. 

[0021] The digital camera 100 incorporates a CCD area sensor 32 (see Fig. 5), and 
captures an image by imaging the light from a capturing object through the taking lens 5 onto 
the sensor 32. The captured image is recorded on the recording medium 6, and is displayed 
on the display 14. An image already recorded on the recording medium 6 can also be 
reproduced and displayed on the display 14. 

[0022] The digital camera 100 has a capability of adding annotation information (an 
annotation) to a region within an image. Hereinafter, this capability will be referred to as the 
annotating capability, and the mode in which the digital camera 100 executes the annotating 
capability will be referred to as the annotation mode. Here, two types of annotation are 
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handled, namely images and sounds. An annotation in the form of an image will be referred 
to as an annotation image, and an annotation in the form of a sound will be referred to as an 
annotation sound. An image to which an annotation is going to be added or has already been 
added will be also referred to as a parent image. 

[0023] The microphone 4 is used to record an annotation sound, and the loudspeaker 19 is 
used to reproduce an annotation sound. In the annotation mode, the operation buttons 15 to 
18 and 20 are operated to select an image, to set a region within the image, and for other 
purposes. The operation buttons 15, 16, 17, 18, and 20 are referred to as the main menu 
button, quick view button, delete button, set button, and region size change button, 
respectively. 

[0024] Now, the annotation capability of the digital camera 100 will be described with 
reference to Figs. 3A and 3B. Fig. 3A shows an example of a parent image having 
annotations added thereto. Annotations are added to regions (of which there may be one or 
more) within the parent image 21. Within the image 21, two regions are set, and the 
corresponding region frames 22 and 23 are displayed. These regions each have an 
annotation added thereto. It should be noted that the region frames 22 and 23 are displayed 
merely for display on the liquid crystal display 14, and are not included in the image 21 itself. 

[0025] Fig. 3B is an example of an annotation image added to the parent image 21. This 
image 24 is a photograph of the father of the girl that appears in the region frame 22. On the 
other hand, to the region frame 23 is added, for example, an annotation sound that speaks 
"this boy is John." 

[0026] The file of the annotation image 24 and the file of the annotation sound are 
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associated with the file of the image 21, and the association among these files is recorded in a 
predetermined file called an association file. When all these files are transferred to a 
personal computer, they can be viewed on the personal computer. This permits better 
understanding of the parent image 21 and better interpersonal communication. 

[0027] Table 1 shows the contents of an association file. 

TABLE 1 

Contents of an Association File 



Item 


Content 


1 


Parent image file name 


2 


Parent image file creation date and time 


3 


Annotation region position (coordinates of the upper left corner) 


4 


Annotation region size (vertical and horizontal dimensions) 


5 


Annotation file name 


6 


Annotation file creation date and time 



[0028] There are as many sets of items 3 to 6 as there are annotations added to the parent 
image. The association file is updated whenever the association information needs to be 
changed, for example when a new annotation is added, when a file is deleted, or in other cases. 

[0029] Fig. 4 shows the structure of file folders in the digital camera 100. In this 
example, the files "Pict0001.jpg" and "Pict0004.jpg" are parent image files. These files are 
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placed in a DCF (digital camera format) folder named "100MLT09," in which there are 
annotation folders "AnnOOOl" and "Ann0004 " The numbers "0001" and "0004" appended 
to the annotation folders are the same as those appended to the respective parent files 
Pict0001.jpg" and "Pict0004.jpg," and this permits association between parent images and 
annotation folders. 

[0030] In each annotation folder, there are stored an association file, annotation image 
files, and annotation sound files. For example, in the folder "AnnOOOl," the file 
"AnnOOOLtxt" is the association file, the files "Ann0001.jpg" and "Ann0002.jpg" are 
annotation image files, and the files "Ann0001.wav" and "Ann0002.wav" are annotation 
sound files. 

[0031] The digital camera 100 has, with respect to capturing of still images, four modes, 
namely a normal photographing mode, a document photographing mode, an image annotation 
photographing mode, and a sound annotation recording mode. The normal photographing 
mode is for ordinary capturing, with the digital camera 100 functioning as a common digital 
camera. The document photographing mode is for capturing a character-based subject such 
as a document or a manuscript on a whiteboard. The image annotation photographing mode 
is for adding an annotation image to a parent image. The sound annotation recording mode 
is for adding an annotation sound to a parent image. The annotation mode mentioned earlier 
thus includes the image annotation photographing mode and the sound annotation recording 
mode. 

[0032] Information relating to in which photographing mode an image or sound was 
captured or recorded is recorded in the tag (header) of the respective file. This 
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photographing mode information is referred to in order to indicate, when an image or sound is 
reproduced and displayed or played back, in which photographing mode the image or sound 
was captured or recorded. The photographing mode information is recorded in a one-byte 
region in the tag. Table 2 shows the individual bits of that region and what a "1" in each of 
them stands for. 

TABLE 2 



Photographing Mode Information 



Bit 


Meaning 


0 


An image captured in the normal photographing mode 


1 


An image captured in the document photographing mode 


2 


A parent image in the image annotation photographing mode 


3 


An annotation image captured in the image annotation photographing mode 


4 


A parent image in the sound annotation recording mode 


5 


An annotation sound recorded in the sound annotation recording mode 


6 


A movie 


7 


A sound other than an annotation sound 



[0033] These bits are updated whenever they need to be changed, for example when an 
image is captured, when an annotation is added, and in other cases. Two or more of the bits 
may be "1" simultaneously. For example, when an image captured in the normal 
photographing mode is used as a parent image in the sound annotation recording mode, the 
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bits 0 and 4 are "1" simultaneously. 

[0034] Fig. 5 schematically shows the circuit configuration of the digital camera 100. 
The output signal of the CCD sensor 32 is converted by a data processor 33 into digital image 
data that represents an image. This image data is displayed on the liquid crystal display 14, 
and is recorded on the recording medium 6 in response to a command to that effect. The 
output signal of the microphone 4 is converted by the data processor 33 into digital sound data. 
This sound data is recorded on the recording medium 6. An image recorded on the 
recording medium 6 can be displayed on the liquid crystal display 14 through the data 
processor 33, and a sound recorded on the recording medium 6 can be output from the 
loudspeaker 19 through the data processor 33. All such operations are controlled by a CPU 
31. 

[0035] Fig. 6 shows the flow of operations performed to set the photographing mode 
when electric power starts to be supplied. The digital camera 100, when the supply of 
electric power thereto is stopped, stores the photographing mode at that moment, and, when 
the supply of electric power thereto is started (step #1), restores the stored photographing 
mode (steps #2 to #6). 

[0036] On the liquid crystal display 14 are displayed, apart from images, various menu 
screens that guide the user through various operations such as switching among the different 
photographing modes. Fig. 7 shows the main menu displayed on the liquid crystal display 
14. The main menu permits selection of one from among the four photographing modes 
mentioned earlier. In the example shown in Fig. 7, the normal photographing mode is 
selected. One of the modes is selected by operation of the four direction keys 13, and is 
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established by operation of the set button 18. This main menu can be reached from 
anywhere by operation of the main menu button 15. 

[0037] Fig. 8 shows the flow of operations performed to change the photographing mode 
in the main menu. The photographing mode is set to the mode that is being selected at the 
moment that the set button 18 is operated (#1 1 to #15). 

[0038] If, in step #3 shown in Fig. 6 or in step #12 shown in Fig. 8, the normal 
photographing mode has been established, the digital camera 100 executes ordinary capturing 
operations as does a common digital camera. On completion of capturing, the digital camera 
100 creates an image file, and registers photographing mode information in its tag. 

[0039] If, in step #4 shown in Fig. 6 or in step #13 shown in Fig. 8, the document 
photographing mode has been established, the digital camera 100 executes ordinary capturing 
operations as does a common digital camera, but does not shoot a flash by default irrespective 
of whether the environment is bright or dim. On completion of capturing, the digital camera 
100 creates an image file, and registers photographing mode information in its tag. 

[0040] Fig. 9 shows the flow of operations performed when, in step #5 shown in Fig. 6 or 
in step #14 shown in Fig. 8, the image annotation photographing mode has been established. 
If this is the first time that this mode is established after the start of the supply of electric 
power (step #21), the association file is updated (#22), and the photographing mode 
information is updated (#23). This is to cope with any updating of the files on the recording 
medium 6 by an external device. On completion of this updating, the digital camera 100 
displays a parent image selection screen (#24). 
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[0041] Fig. 10 shows an example of the parent image selection screen. In the parent 
image selection screen, the images and sounds stored in the files recorded on the recording 
medium 6 are displayed as thumbnail images. Here, since sounds themselves cannot be 
displayed, for a sound file, a thumbnail image 43 of a musical note is displayed. The user 
selects a parent image from among the images displayed. In the example shown in Fig 10, 
the image 41 of which the frame is indicated with a thick line is being selected. One of the 
images is selected by operation of the four direction keys 13, and is established by operation 
of the set button 18. 

[0042] The symbol "Al" displayed contiguous with the thumbnail image 41 indicates that 
the image of the corresponding file has already been used as a parent image in the image 
annotation photographing mode. On the other hand, the symbol "A2" displayed contiguous 
with the thumbnail image 44 indicates that the image of the corresponding file has already 
been used as a parent image in the sound annotation recording mode. In this way, the 
properties of images are displayed on the basis of their photographing mode information to 
facilitate selection of a parent image. Incidentally, the symbol "M" displayed overlapping 
the thumbnail image 42 indicates that the image of the corresponding file is a movie. 

[0043] Fig. 1 1 shows the flow of operations performed to select a parent image. First, 
thumbnail images are displayed (step #41). If these images include the one that was selected 
as a parent image last time, that image is displayed in a selected state; if not, the image that 
was captured most recently is displayed in a selected state (#42 to #44). The user is likely to 
select as a parent image the one that the user selected as a parent image last time. 
Accordingly, by displaying in a selected state the image that was selected as a parent image 
last time, it is possible to alleviate the operation for selection of a parent image. 
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[0044] To minimize unnecessary power consumption, the digital camera 100 has an 
automatic power-off capability, whereby the supply of electric power is automatically stopped 
when the user makes no operation for a predetermined period of time. Thus, the automatic 
power-off capability may work while the user is suspending operation after selecting a parent 
image. Even in that case, the user can continue with operation without newly selecting a 
parent image. Here, the image that was captured most recently may be displayed in a 
selected state regardless of whether or not there is any image that was selected as a parent 
image last time. 

[0045] Alternatively, instead of the image that was selected as a parent image last time, an 
image to which an annotation has already been added may be displayed in a selected state. 
The user is likely to add a new annotation to an image to which an annotation has already 
been added. Accordingly, by displaying such an image in a selected state, it is possible to 
alleviate the operation for selection of a parent image. In that case, among images to which 
annotations have already been added, the one that was captured most recently may be 
displayed in a selected state, or the one to which the newest annotation has been added may 
be displayed in a selected state. 

[0046] Next, one of the images displayed is selected by operation of the direction keys 13, 
and the one that is being selected at the moment that the set button 18 is operated is 
determined as a parent image (#45). Then, whether the determined image is usable as a 
parent image or not is checked (#46 to #48). Specifically, it is checked whether or not the 
maximum number of annotations that can be added to a parent image will be exceeded by 
adding a new annotation (in the example being discussed, the maximum number is ten), and 
whether or not the selected file is that of a movie or sound. If one of these applies, a 
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warning as shown in Fig. 12 is displayed (#49), and the flow then returns to step #45. 

[0047] On completion of selection of a parent image, a region setting screen is displayed 
that permits setting of the position and size of a region within the parent image to which to 
add an annotation image (Fig. 9, step #25). Fig. 13 shows an example of the initial state of 
this region setting screen. In the region setting screen, along with the selected parent image 
is displayed a default region frame 5 1 located at the center of the image. The user moves 
this region frame 51 to a position where the user wants to add an annotation. The region 
frame 51 is moved by operation of the four direction keys 13. When the user wants to 
change the size of the region to which to add an annotation, the user enlarges or reduces the 
region frame 5 1 . The region frame 5 1 is enlarged or reduced by operation of the region size 
change button 20. Fig. 14 shows an example of the region setting screen including a region 
frame 5 1 having its position changed, and Fig. 1 5 shows an example of the region setting 
screen including a region frame 53 having both its position and size changed. 

[0048] The region that is displayed in the region setting screen by default may be the 
region that was used for focus adjustment when the parent image was captured. The digital 
camera 100 adopts servo-type automatic focus adjustment whereby the focus of the taking 
lens 5 is varied in the direction in which increasingly high contrast is obtained until the focus 
is located at where the maximum contrast is obtained. As a region in which to detect 
contrast, either a whole image or part thereof can be selected, and, as part of an image, one of 
a plurality of different regions within the image, including one located at the center thereof, 
can be selected. By using as a default region the one among those regions which was 
actually used to adjust focus when the parent image was captured, it is possible to easily add 
an annotation to the part of an image which was of particular interest to the user when the user 
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captured it. The region that was used for focus adjustment is registered in the tag of the 
image file. 

[0049] Fig. 16 shows the region setting screen displayed in a case where an annotation 
has already been added to the selected parent image. The region frame 54 corresponds to the 
annotation that has already been added, and the region frame 55 is the one displayed by 
default to permit a new annotation to be added. The region frames 54 and 55 are displayed 
in different manners, such as in different colors, different line types, or different line 
thicknesses. When two or more annotations have already been added, the region frames 
corresponding to the individual annotations are all displayed in different manners. 

[0050] Fig. 17 shows the flow of operations performed to set a region. First, if there is 
any already added annotation, its region frame is displayed (step #51), and then a default 
region frame for an annotation to be newly added is displayed (#52). Then, in response to 
the user's operation, the desired position and size of the region are determined. 

[0051] Now, back in Fig. 9, the flow of operations in the image annotation photographing 
mode continues as follows. After the region setting screen is displayed and a region to 
which to add an annotation is set therein, an annotation image is captured (step #26), then the 
association file is updated (#27), and then the photographing mode information is updated 
(#28). Then, whether or not the total number of annotations added to the parent image has 
reached the maximum number (here, as described earlier, ten) is checked (#29). If the total 
number has reached the maximum number, the flow returns to step #24 to permit selection of 
a new parent image; if not, the flow returns to step #25 to permit setting of a region within the 
same parent image to which to add a new annotation. 
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[0052] When the user wants to select a new parent image before the total number of 
annotations reaches ten, the user can operate the main menu button 15 to return from step #25 
to step #24. When the main menu button 15 is operated in step #24, the main menu 
described earlier, i.e., the menu that permits selection of the photographing mode, is displayed. 

[0053] Fig. 18 shows the flow of operations performed when, in step #5 shown in Fig. 6 
or in step #14 shown in Fig. 8, the sound annotation recording mode has been established. 
Here, in place of the step of capturing an annotation image (step #26), there is inserted a step 
of recording an annotation sound (step #26S). 

[0054] The updating of the association file and of the photographing mode information is 
executed at the time points (1) to (4) described below according to the rules described below. 

[0055] (1) On completion of capturing of an annotation image or recording of an 
annotation sound. 

a) If no annotation has already been added to the parent image, a new annotation 
folder and a new association file are created, and annotation information is registered in 
the association file. Moreover, the photographing mode information of the parent image 
file is updated, and photographing mode information is registered in the created 
annotation image or sound file. 

b) If any annotation has already been added to the parent image, new annotation 
information is additionally registered in the association file. Moreover, the 
photographing mode information of the parent image file is updated, and photographing 
mode information is registered in the created annotation image or sound file. 
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[0056] (2) When the image annotation photographing mode or sound annotation 
recording mode is established for the first time after the start of the supply of electric power. 

a) For a given annotation folder, whether the parent image file, annotation image 
files, and annotation sound files registered in the association file stored therein exist in 
that annotation folder or not is checked. Then, portions of the association file which 
relate to any nonexistent file are deleted. 

b) For a given annotation folder, if none of the annotation image files and 
annotation sound files registered in the association file stored therein exists, that 
association file and the annotation folder are deleted. 

c) For a given annotation folder, if the parent image file registered in the 
association file stored therein does not exist, that association file and the annotation folder 
(along with the annotation image files and annotation sound files stored therein) are 
deleted. 

d) For a given annotation folder, if no association file exists therein, the 
annotation folder (along with the annotation image files and annotation sound files stored 
therein) is deleted. 

e) The photographing mode information of the parent file, annotation image files, 
and annotation sound files is updated so as to be consistent with the operations a) to d) 
above. 

f) The operations a) to e) above are executed for all annotation folders. 
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[0057J (3) When the recording medium is mounted while electric power is being supplied. 

The same operations as described under (2) above are executed. 
[0058] (4) When an external device is disconnected while electric power is being supplied. 

The same operations as described under (2) above are executed. 

[0059] Fig. 19 shows the flow of operations described under (1) above, Fig. 20 shows the 
flow of operations described under (2) above, Fig. 21 shows the flow of operations described 
under (3) above, and Fig. 22 shows the flow of operations described under (4) above. 

[0060] The digital camera 100 has a capability of providing a quick view, whereby an 
image or sound recorded on the recording medium 6 is reproduced in the photographing mode. 
The quick view capability can be used in any of the normal photographing, document 
photographing, image annotation photographing, and sound annotation recording modes. A 
quick view is started by operation of the quick view button 16. 

[0061] Fig. 23 shows the flow of operations performed to provide a quick view. When 
the quick view button 16 is operated, the image, movie, or sound that was captured or 
recorded most recently is reproduced (step #81). During reproduction, by operating, among 
the four direction keys 13, the left or right key, it is possible to switch the reproduced image 
or sound. During reproduction, by operating the delete button 17 (#82), it is possible to 
delete the file of the image or sound that is being reproduced (#83). 

[0062] When the quick view button 16 is operated again, the quick view is ended (#84). 
When a quick view is ended, the screen that was displayed before the quick view was started 
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is displayed again except if it was started when the region setting screen (Fig. 9, step #25) was 
being displayed. If the quick view was started when the region setting screen was being 
displayed, first, whether the parent image in which a region was then going to be set was 
thereafter deleted during the reproduction of the quick view or not is checked (#86). Then, if 
that image has not been deleted, the original region setting screen is displayed again; if that 
image has been deleted, there is no longer any sense in selecting that parent image, and 
therefore the parent image selection screen (Fig 9, step #24) is displayed instead. 

[0063] Fig. 24 shows the flow of operations performed to delete a file in step #83. When 
the delete button 17 is operated, the file of the image, movie, or sound that is being 
reproduced is deleted (step #91). Then, whether or not the deleted file is that of a parent 
image, annotation image, or annotation sound is checked (#92). If this applies, the 
association file and the photographing mode information are updated according to Tables 1 
and 2 (#93 and #94). 

[0064] Fig. 25 shows another flow of operations performed to start a quick view when the 
region setting screen is being displayed. Through this flow, only the annotation images and 
annotation sounds added to the selected parent image are reproduced (step #96). Likewise, 
only the annotation images and annotation sounds added to the selected parent image can be 
deleted. This makes it easy to grasp what annotations are added to the parent image, and 
helps reduce the risk of erroneously deleting files that are unrelating to the parent image. 

[0065] The annotation mode of the digital camera 100 thus far described consists of 
separate flows of operations for the image annotation photographing and sound annotation 
recording modes, of which one is selected in the main menu (Fig. 7). This configuration is 
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convenient in a case where a plurality of annotation images are successively added to a parent 
image or a plurality of annotation sounds are successively added to a parent image. 
However, this is inconvenient in a case where both an annotation image and an annotation 
sound are added to a parent image, because it is then necessary to return to the main menu to 
switch the photographing mode. This requires somewhat troublesome operation. 

[0066] Moreover, the method described thus far permits annotations to be added only to 
images that have already been recorded on the recording medium 6. Thus, when the user 
wants to capture an image first and then add an annotation thereto, the user needs to go to the 
main menu first to establish the normal photographing mode, then capture and record an 
image, then go to the main menu again to establish the image annotation photographing or 
sound annotation recording mode, and then capture an annotation image or record an 
annotation sound. This, too, requires somewhat troublesome operation. 

[0067] Now, a description will be given of a method of efficiently setting the 
photographing mode in a case where different types (image and sound) of annotations are 
successively added and in a case where an annotation is added to an image that is going to be 
captured. Fig. 26 shows the main menu displayed when this method is used. Instead of 
separate image annotation photographing and sound annotation recording modes, here a 
single annotation photographing mode is provided, and is displayed parallel with the normal 
photographing and document photographing modes. 

[0068] Fig. 27 shows the flow of operations performed in the annotation photographing 
mode. First, if this is the first time that this mode is established after the start of the supply 
of electric power, the association file and the photographing mode information are updated 
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(steps #62 to #64). These steps are the same as steps #21 to #23 shown in Fig. 9 described 
earlier. Next, the screen shown in Fig. 28 is displayed to permit the user to choose whether 
to use as a parent image an image that is going to be captured or an image that has already 
been recorded on the recording medium 6 (step #65). The user selects the desired choice by 
operating the direction keys 13, and then determines the choice by operating the set button 18. 

[0069] If an image that is going to be captured is chosen as a parent image, the flow 
proceeds to step #67; if an image that has already been recorded is chosen as a parent image, 
the flow proceeds to step #66. In step #66, the flow of operations shown in Fig. 1 1 
described earlier is executed. In step #67, an image that is to be used as a parent image is 
captured just as in ordinary capturing of an image. 

[0070] When the parent image is determined in this way, then, as in step #25 shown in Fig. 
9, the region setting screen is displayed to permit setting of the position and size of the region 
to which to add an annotation (#68). After the region is set, the screen shown in Fig. 29 is 
displayed to permit the user to choose whether to use as an annotation an image or sound 
(#69). The user selects the desired choice by operating the direction keys 13, and determines 
the choice by operating the set button 18. 

[0071] According to the choice made, an annotation image is captured (#70), or an 
annotation sound is recorded (#71). Then, as in steps #27 and #28 shown in Fig. 9, the 
association file and the photographing mode information are updated (steps #72 and #73). 
Then, whether the total number of annotations added to the parent image has reached the 
maximum number (here, ten) or not is checked (#74). If the maximum number has been 
reached, the flow returns to step #65 to permit the user to choose whether to use as a parent 
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image an image that has already been recorded or an image that is going to be captured; if the 
maximum number has not been reached, the flow returns to step #68 to permit setting of the 
position and size of a region. 

[0072] In this method of setting the photographing mode also, when the supply of electric 
power is stopped, the photographing mode at that moment is stored so that, when the supply 
of electric power is started next time, the stored photographing mode is restored. Fig. 30 
shows the flow of operations performed to set the photographing mode when electric power 
starts to be supplied. Fig. 31 shows the flow of operations performed to change the 
photographing mode in the main menu. 

[0073] This method permits efficient setting of the photographing mode in the following 
two aspects. When the user wants to add an annotation to an image that is going to be 
captured, the user can capture the image first and then immediately add an annotation thereto. 
In a case where an annotation image and an annotation sound are successively added to a 
single parent image, there is no need to select the parent image in the middle of the necessary 
operation. 

[0074] Here, the screen that permits choice of whether to use as a parent image an image 
that is going to be captured or an image that has already been recorded on the recording 
medium 6 and the screen that permits selection of which of the images already recorded on 
the recording medium 6 to use are displayed successively in different steps (Fig. 27, steps #65 
and #66). These two screens may be merged into one. Fig. 32 shows an example of the 
screen displayed in that case. 

[0075] This screen, like the one shown in Fig. 10 described earlier, displays in an 
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arranged manner the thumbnail images of files recorded on the recording medium 6, and in 
addition displays an image 45, marked "capture a parent image," that permits an image that is 
going to be captured to be used as a parent image. The user, by selecting this image 45, can 
request that an image that is going to be captured be used as a parent image, and, by selecting 
one of the thumbnail images other than the image 45, can request that an already recorded 
image be used as a parent image and simultaneously specify which of the already recorded 
images to use as a parent image. This helps reduce the number of times that the screen 
needs to be switched, and thus helps further increase efficiency. 

[0076] Obviously, many modifications and variations of the present invention are possible in 
light of the above teachings. It is therefore to be understood that within the scope of the 
appended claims, the invention may be practiced other than as specifically described. 
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